Don ’ t touch a word ! A practical input eavesdropping attack against mobile touchscreen devices

Spying on a person is a subtle, yet easy and reliable method to obtain sensitive information. Even if the victim is well protected from digital attacks, spying may be a viable option. In addition, the pervasiveness of mobile devices increases an attacker’s opportunities to observe the victims while they are accessing or entering sensitive information. This risk is exacerbated by the remarkable user-friendliness of modern, mobile graphical interfaces, which, for example, display visual feedback to improve the user experience and make common tasks, e.g., typing, more natural. Unfortunately, this turns into the well-known trade-off between usability and security. In this work, we focus on how usability of modern mobile interfaces may affect the users’ privacy. In particular, we describe a practical eavesdropping attack, able to recognize the sequence of keystrokes from a low-resolution video, recorded while the victim is typing on a touchscreen. Our attack exploits the fact that modern virtual keyboards, as opposed to mechanical ones, often display magnified, virtual keys in predictable positions. To demonstrate the feasibility of this attack we implemented it against 2010’s most popular smart-phone, i.e., the iPhone. Our approach works under realistic conditions, because it tracks and rectifies the target screen according to the victim’s natural movements, before performing the keystroke recognition. On real-world settings, our attack can automatically recognize up to 97.07% (91.03% on average) of the keystrokes, with a 1.15% error rate and a speed between 37 and 51 keystrokes per minute. This work confirms that touchscreen keyboards that magnify keys make automatic eavesdropping attacks easier than in classic mobile keyboards.